Skip to content

Add Smithsonian fetch#257

Merged
TimidRobot merged 5 commits intomainfrom
smithsonian
Jan 6, 2026
Merged

Add Smithsonian fetch#257
TimidRobot merged 5 commits intomainfrom
smithsonian

Conversation

@TimidRobot
Copy link
Member

@TimidRobot TimidRobot commented Dec 16, 2025

Fixes

Description

Add Smithsonian fetch script

Technical details

Example smithsonian_1_metrics.csv (click to expand)
CC0_RECORDS CC0_RECORDS_WITH_CC0_MEDIA CC0_MEDIA CC0_MEDIA_PERCENTAGE TOTAL_OBJECTS
14273329 5199915 4503016 36 15616799

Table created with command and manually added deliminator row:

sed -e's/","/ | /g' -e's/^"/| /' -e's/"$/ |/' data/2025Q4/1-fetch/smithsonian_1_metrics.csv
Example smithsonian_2_units.csv (click to expand)
UNIT CC0_RECORDS CC0_RECORDS_WITH_CC0_MEDIA TOTAL_OBJECTS
AAA 0 0 29735
AAG 0 0 344
ACM 251 247 2977
ACMA 0 0 57
CFCHFOLKLIFE 17544 0 18517
CHNDM 58158 54590 201545
FBR 1517 37 11248
FSG 4720 4720 45588
HAC 430 430 1437
HMSG 449 448 13898
HSFA 0 0 299
NASM 1010 989 32325
NMAAHC 22224 4465 22577
NMAH 1316502 10548 1317248
NMAI 237637 180 239307
NMAfA 111 111 12477
NMNHANTHRO 497734 0 497734
NMNHBIRDS 635217 559038 635217
NMNHBOTANY 4562256 3572487 4562256
NMNHEDUCATION 6473 4090 6473
NMNHENTO 731838 197223 731838
NMNHFISHES 502585 10806 502585
NMNHHERPS 615308 2345 615308
NMNHINV 2003972 70094 2003972
NMNHMAMMALS 626133 542046 626133
NMNHMINSCI 465275 11311 465275
NMNHPALEO 743533 94487 743533
NPG 15446 14540 123566
NPM 10814 8005 83710
NZP 1061 1061 2086
OCIO_DPO3D 108 17 146
OFEO-SG 5509 3665 7295
SAAM 13626 12891 188157
SIA 35498 5477 48169
SIL 1035579 13567 1039087
SILAF 63416 0 63416
SILNMAHTL 34577 0 34577
SLA_SRO 104811 0 104811

Table created with command and manually added deliminator row:

sed -e's/","/ | /g' -e's/^"/| /' -e's/"$/ |/' data/2025Q4/1-fetch/smithsonian_2_units.csv

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated unit tests and/or test scripts for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin 
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@TimidRobot TimidRobot self-assigned this Dec 16, 2025
@TimidRobot TimidRobot requested review from a team as code owners December 16, 2025 08:34
@TimidRobot TimidRobot requested review from Shafiya-Heena and removed request for a team December 16, 2025 08:34
@github-project-automation github-project-automation bot moved this to Triage in TimidRobot Dec 16, 2025
@TimidRobot TimidRobot moved this from Triage to In review in TimidRobot Dec 16, 2025
@TimidRobot TimidRobot requested review from oree-xx and removed request for Shafiya-Heena December 17, 2025 12:24
oree-xx
oree-xx reviewed Dec 18, 2025
View reviewed changes
Copy link
Contributor

@oree-xx oree-xx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me!

@TimidRobot TimidRobot requested a review from oree-xx December 22, 2025 14:08
oree-xx
oree-xx requested changes Jan 5, 2026
View reviewed changes
Copy link
Contributor

@oree-xx oree-xx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be LOGGER.info("Fetching media and record counts from internet archive") at line 131.

@TimidRobot TimidRobot requested a review from oree-xx January 5, 2026 18:07
oree-xx
oree-xx approved these changes Jan 6, 2026
View reviewed changes
Copy link
Contributor

@oree-xx oree-xx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work! This is good to go.

@TimidRobot TimidRobot merged commit 94a193a into main Jan 6, 2026
2 checks passed
@TimidRobot TimidRobot deleted the smithsonian branch January 6, 2026 17:11
@github-project-automation github-project-automation bot moved this from In review to Done in TimidRobot Jan 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@oree-xx oree-xx oree-xx approved these changes

@possumbilities possumbilities Awaiting requested review from possumbilities

Assignees

@TimidRobot TimidRobot

Labels

None yet

Projects

TimidRobot
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Smithsonian Institution Archives as data source

2 participants

@TimidRobot @oree-xx